ALLEGHENY COUNTY DEPARTMENT OF HUMAN SERVICES 

Using Predictive Modeling 
to Improve Outcomes 

For Children in Allegheny County 


Training for Child Welfare 
Professionals 




Agenda 


• Objectives 

• Overview of Predictive Analytics/Predictive Risk Modelling 

• Application of Predictive Analytics in Allegheny County 

• Policy/Practice 

• Case scenarios 

• Process changes 

• KIDS Changes/Design 

• Ouestions 
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Objectives 


By the end of this training, participants wiii be abie to... 

• Understand the basics of Predictive Analytics/Predictive Risk Modeling 

• Describe how the Family Screening Score will be generated in KIDS 

• Understand what the Family Screening Score means 

• Explain how the Family Screening Score is to be included in making screening decisions and the policy 
for its use 

• Understand the addition of dynamic text from Client View to a referral in KIDS 
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I What is Predictive 
Risk Modeling? 




What is Predictive Risk Modeling? 


• Use data already collected about each child to evaluate the possibility of future 
adverse outcomes 

• Why consider predictive risk modeling? 

- Wider availability of high quality information about service interactions that can inform our 
understanding of risk and protective factors 

- Advances in technology and analytic capabilities- we can use these data in real time rather than 
retrospective analyses 

- Difficulty in weighting complex factors (and time is scarce!) 
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I How is Predictive Analytics 
Being Applied in Allegheny 
County? 




Developing a Screening Score 


• We developed a partnership with a research 
team based in New Zealand 

• We shared Allegheny County data with the 
research team to analyze for factors that would 
be predictive of future maltreatment 

• Call screening was selected as the decision point 
at which a model would be developed and used 

• The team developed a model that includes more 
than 100 pieces of information 

• This screening score is comprised of the 
likelihood of re-referral, given screen-out and the 
likelihood of placement in foster care, given 
screen-in 
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Running a Screening Score 


• The tool is called the Allegheny Family Screening Tool 

• The score that will be produced is called the Family Screening Score 

• The score will be run in the KIDS system which will harvest data from the data warehouse* 

• At each call, a screening score will be run for each child associated with an allegation of 
maltreatment 

• Only one score will be displayed to the screener- the highest score for the highest child on that 
referral 

• The score only applies at call screening 

• The score will only be viewed by call screeners and call screening supervisors 

- It will not be seen by investigators or ongoing workers and it is expected that call screeners will not share the score with 
others 
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* Only if an MCI_ID is successfully established for at least one HHM 
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Developing a Screening Score 


• The screening score is from 1 to 20 (with a subset of referrals being mandatory screen ins) 

• The higher the score, the higher the chance of the future event 

(e.g., abuse, placement, re-referral) according to the data 
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But will it work? Will it actually give me 
a score that reflect this family's likelihood of 
future maltreatment? 

To test that, we scored thousands of 
historical maltreatment calls and then 
followed the children in subsequent 
referrals to see how often the model was 
correct... 
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Developing the model 



24,188 

Index Referrals 


r-@53% ( 12 , 894 ) 

I Screened Out 


r--© 35% ( 4 , 467 ) 

! Re-referred within a year 

j 

i-065% ( 8 , 418 ) 

No Re-referrals 




047% ( 11 , 214 ) 


Screened In 


r-©24% ( 2 , 681 ) 

! Re-referred within a year 

•i 

I 

I 

^--© 76 % ( 8 , 533 ) 

No Re-referrals 
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The 



1 in 10 children 


with a score of 1 were re-referred 

within two years of the call. 
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: Re-Referrals 


JUlfWAll 

9 in 10 children 

with a score of 20 were re-referred 

within two years of the call. 
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The Results: 

Out-of-Home Placements 





1 in 100 children. 

who received a score of 1 were placed 
out-of-home within 2 years of the call 
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The Results: 

Out-of-Home Placements 




1 in 2 children 


who received 

a score of 20 were placed 
out-of-home within 
2 years of the call 
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Under current practice: 


27% of highest risk cases 
were screened out — 

of these, 1 in 3 are re-referred 
and placed within 2 years of the 
initial screened out call 

48% of lowest risk cases 
were screened in — 

only 1.4% of those 
are placed within 2 years. 
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Approaches to Assessing Risk 



PA Risk Assessment 

Family Screening Score 

Definition 

Likelihood of future maltreatment 

Likelihood of future re-referral or placement 

Based on 

Clinical judgement of worker 

Data driven 

Timeframe 

Future 

Within two years of call 

Type assessed 

Acute and chronic 

Chroniconly 

Considers 

All aspects of the family life relevantto 
future maltreatment (allegations,family 
stressors, prior history) 

All factors found to be predictive through 
modeling of all Data Warehouse variables for 
CYF referrals 2010 -2013 

Includes 

15 factors 

Over 100 factors 

Applies 

At all critical child welfare decision points 
and when new information arises 

Only at call screening decision 

Output 

Overall Severity: seriousness of what has 
happened 

Overall Risk: likelihood of future harm 

Flighest score for highest child for likelihood of 
either re-referral or placement 

























I What is the Policy and 
Process for 
Using the Family 
Screening Score? 



Policy 


The Allegheny County Office of Children, Youth, and 
Families (OCYF) Call Screening Department will 
employ the Allegheny Family Screening Tool as one of 
the tools used in determining a screening decision for 

all incoming referrals. 
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Outcome 


Call Screener Findings 


Ris k Rating * _ 

[low Ris k (L) I 


Sa fety Rating* 
[no Safety Threat 


Basis for Risk Rating * 


ufyfyfty 

V 

Basis for Safety Rating* 

ugugugugu 

V 


Please review the family screening score after entering the risk and safety ratings * 
Family Screening Score 


Current Screening Score: 



Last Run Date: 
03/12/2018 08:59 AM ' 


Call Screener Recommervdation * 

Accept for Investigation v} 


Predisposition 


Assign GPS Investigation to 
Active Caseworker 


O Assign to Field Screening Unit 


Field Scr c etimQ Unit Recomn>erHiatk>n 


Referral/Case ID 


find 


lteason(s) for assigning to Field Scr ee ning Unit 


Reason for Recommendation 

\P 


Select 


Disposition 


^Accept Open Investigation 

C) Screen Out I 


Response Time* 




0 Immediate 

O Priority (Within 24 hours) 


i9:52 

DpiicaiionuasnDoara.riTmi 


o] e ■ ^ ^ % 


June 27, 2016 
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Now we know what it is, we 
know what it will look like, 
we know how to run it. 

But, how does it impact our 
decision making? 

Let's look at some case 
scenarios... 



Case Scenario 1 


• Referral date: 4/2015 





Case 


You said we should: 


The screening score was: 


Does that change your decision? 


1 
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Referral date: 11/2015 




I 



I 
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Case 


You said we should: 


The screening score was: 


Does that change your decision? 
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Case Scenario 4 


• Referral date: 3/2014 
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Case 


You said we should: 


The screening score was: 


Does that change your decision? 
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Case Scenario 6 
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Case 


You said we should: 


The screening score was: 


Does that change your decision? 
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Questions? 




